DARPA invests in AI that can translate instruction manuals into augmented reality

WASHINGTON – The Defense Advanced Research Projects Agency has issued a $5.8 million contract to a team building an artificial intelligence system able to scan instruction manuals and convert that data into instructions for augmented reality systems.

Companies are already using augmented reality technologies in their manufacturing processes. Lockheed Martin, for example, uses augmented reality goggles in assembling its space systems for NASA. With the goggles on, technicians can see relevant information and instructions in the space around them as they go about their work, saving them from having to constantly walk back and forth to consult physical manuals or computer monitors.

Under the $5.8 million contract, PARC, a Xerox company, will work with the University of California at Santa Barbara, the University of Rostock in Germany and Patched Reality on the Autonomous Multimodal Ingestion for Goal-Oriented Support (AMIGOS) project for the Perceptually-enabled Task Guidance Program. In short, the goal is to take the existing paper and video manuals used today and automatically convert them for use in augmented reality systems.

“Augmented reality, computer vision, language processing, dialogue processing and reasoning are all AI technologies that have disrupted a variety of industries individually but never in such a coordinated and synergistic fashion,” said Charles Ortiz, the principal investigator for AMIGOS, in a statement. “By leveraging existing instructional materials to create new AR guidance, the AMIGOS project stands to accelerate this movement, making real-time task guidance and feedback available on-demand.”

The teams will deliver two different but related systems to DARPA. The first is an artificial intelligence system that will be able to extract task information from texts, illustrations, and videos. The second system will take that information and create an augmented reality guidance based on it. Moreover, the second AI will be able to deliver tasks and information in a personalized way based on the user’s skills and emotional state.

PARC is the lead contractor on the project.

More In Artificial Intelligence

New US Space Force jammers aim to disrupt China’s SATCOM signals

The Remote Modular Terminals, which disrupt adversary satellites by "yelling in their ear," as one official put it, are cleared for initial fielding,

Space Force must grow to counter China and Russia, lawmaker says

House Armed Services Chairman Mike Rogers, R-Ala., said the service is not sized to meet increasing threats from China and Russia.

SpaceX launches rapid response GPS mission

The effort, dubbed Rapid Response Trailblazer, was meant to demonstrate the ability to quickly plan and launch a mission in around six months.

Space Command strategy aims to boost commercial role in operations

The updated strategy, approved in November, focuses on SPACECOM’s core tasks and how commercial space capabilities fit within those areas.

German Navy tests BlueWhale underwater drone for covert ship tracking

“This a capability currently only available via manned submarines, which makes it significant for maritime surveillance,” officials said.